Lecture 4 

Image Segmentation 


Dr. Amira Salah Ashour 


Image analysis and segmentation 


• Detection of Discontinuity 

■ Point, line, edge and combined detection.. 

• Edge linking and boundary detection 

■ Local processing, hough transform, graph-theoretic technique. 

• Thresholding 

■ Global thresholding. Optimal thresholding, threshold selection 

• Region oriented segmentation 

■ Region growing, Region splitting and merging.. 


Introduction 

• Image analysis :- 

Techniques for extracting information from an image. 

• Segmentation is the first step for image analysis. 

• Segmentation is used to subdivide an image into its constituent parts or 
objects. 

• Segmentation: 

- Split or separate an image into regions 

- To facilitate recognition, understanding, and region of interests (ROI) processing 

• This step determines the eventual success or failure of image analysis. 

• Generally, the segmentation is carried out only up to the objects of interest 
are isolated, e.g. face detection. 

• The goal of segmentation is to simplify and/or change the representation of an 
image into something that is more meaningful and easier to analyze. 



Segmentation 

• Image segmentation is the process of 
partitioning a digital image into multiple 
parts 


• It divide the image into: 

meaningful and/or perceptually uniform 
regions 

• Segmentation is typically used to locate 

objects and boundaries of physical 
entities in the scene 

• The segmentation process utilizes 

available image information (graylevel, 
colour, texture, pixel position, ...). 
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Segmentation methods 


Image Segmentation: 

• Thresholding techniques 

• Clustering methods for segmentation 

• Morphological operations 


Active contours (Snakes, Scissors, Level Sets) 

Split and merge (Watershed, Divisive & agglomerative clustering, Graph-based 
segmentation) 

K-means (parametric clustering) 

Mean shift (non-parametric clustering) 

Normalized cuts 
Graph cuts 

Graylevel thresholding 


Classification of the Segmentation techniques 


Image Segmentation 
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Discontinuity 


e.g. 

- Point Detection 

- Line Detection 

- Edge Detection 


Similarity 



e.g. 

-Thresholding 

- Region Growing 

- Region splitting & 

merging 











Point Detection 


Apply detection mask, followed by threshold detection 
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FIGURE 10.2 

(a) Point 
detection mask. 

(b) X-ray image 
of a turbine blade 
with a porosity. 

(c) Result of point 
detection. 

(d) Result of 
using Eq. i;10.1-2). 
(Original image 
courtesy of 
X-TEK Systems 
Ltd.) 









Point Detection 


• It is based on Masking 
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• Find response R (response of convolution). 

• The importance is strictly to detect points (isolated points). 

• Compare and separate based on 



T = non-negative threshold value 











Point Detection(Example) 


Original Image after applying mask 






Line Detection 
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Horizontal +45° 
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Vertical —45° 

Useful for detecting lines 
with width = 1. 
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FIGURE 10.4 

Illustration of line 
detection. 

(a) Binary wire- 
bond mask. 

(b) Absolute 
value of result 
after processing 
with —45° line 
detector. 

(c) Result of 
thresholding 
image (b). 




























Line Detection 


Horizontal Line 
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Vertical Line 
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45 degree inclined Line 
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-45 degree inclined Line 
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Line Detection(Cont.) 

Original Image Horizontal line detection 




45 degree inclined line detection 



Vertical line detection 



135 degree inclined line detection 









Edge Detection 

Points and lines are special cases of edges. 

Edge detection is difficult since it is not clear what amounts to 
edge! 

a b 

FIGURE 10.5 

(a) Model of an 
ideal digital edge. 

(b) Model of a 
ramp edge. The 
slope of the ramp 
is proportional to 
the degree of 
blurring in the 
edge. 


Model of an ideal digital edge 



Gray-level profile 
of a horizontal line 
through the image 


Model of a ramp digital edge 




Gray-level profile 
of a horizontal line 
through the image 












Edge Detection 

Robert's Mask 


Prewitt Operator 
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Laplacian 
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Edge Detection(Example) 

Original Image 



Sobel-Horizontal Edge 


Sobel-Vertical Edge 


Sobel-Edge 

















Edge Detection(Example) 


Original 


Laplacian to unsharped image 


After Thresholding by 15% 


















Combined Detection 

Multimask formulation makes possible development of a method to 
determine whether a pixel is most likely to be an isolated point or part 
of a line or an edge . 






Edge Linking and Boundary Detection 

• Intensity discontinuity can be utilized to find boundary. 


• The lagging part of boundary detection using intensity 
discontinuity is that the boundary may not be completely defined 
because of 

• Noise 

• Breaks in boundary due to non-uniform illumination 


So, after edge detection, edge linking process is carried out to 
assemble edge pixels into meaningful boundary 
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Boundary Extraction 


• Edge detection classifies individual pixels 
to be on an edge or not. 

• Isolated edge pixels is more likely to be 
noise rather than a true edge. 

• Adjacent or connected edge pixels 
should be linked together to form 
boundary of regions that segment the 
image. 


Edge linking methods 

- Local processing 

- Hough transform 

- Graphic theoretic 
method 

- Dynamic 
programming 


Threshold Segmentation 



T 



a b 

FIGURE 10.2-6 (a) Gray-level histograms that can be partitioned by (a) a single thresh¬ 
old, and (b) multiple thresholds. 





































Colour Segmentation - Example 


Original imaga 











Segmentation by thresholding 


Number of pixels 



Graylevel 


Otsu’s method: 

• Automatic clustering-based thresholding 

• Minimization of intra-class variance 

• Analog to Fisher’s Discriminant Analysis 








Basic Global Thresholding 


1. Select an initial estimate for the global threshold,!. 

2. Segment the image using T. It will produce two groups of pixels: G1 
consisting of all pixels with intensity values >T and G2 consisting of 
pixels with values <T. 

3. Compute the average intensity values ml and m2 forthe pixels in Gi 
and G2, respectively. 

4. Compute a new threshold value. 

T = ^-(ml + m2) 

5. Repeat Steps 2 through 4 until the difference between values of T in 
successive iterations is smallerthan a predefined parameter AT 



a b c 

FIGURE 1 0.38 (a) Noisy fingerprint, (b) Histogram, (c) Segmented result using a global threshold (the border 
was added for clarity). (Original courtesy of the National Institute of Standards and Technology.) 





















































sing Otsu's 

Method 

* Principle: maximizing the between-class variance 

Let {0. I, 2. .... L-1} denote the L distinct intensity levels 
in a digital image of size M x N pixels, and let n t denote the 
number of pixels with intensity i. 

L - 1 

p } = n ; / MN and T" p } = 1 

j=0 

k is a threshold value, C. —» [0,/r], C 2 —> [/c + l,L - i] 

alld P 2( k )= 2 Pt = l - p xi k ) 

1=0 i=fc+l 


Optimum Global Thresholding U 

‘rVVVVVVVVVVVVV'V'VVVVVVVVVVVVVVVVWVHhjiV 




Optimum Global Thresholding Using Otsu's 
Method 


The mean intensity value of the pixels assigned to class 

Cj is 

m i (k) = T iP(i fC l ) = £ & 

r=0 M t/H f=0 

The mean intensity value of the pixels assigned to class 


C ? is 


Z-l 


(£) = ^ zP(z / C : ) = 


1 


i -1 


Z 


?=£+! 


■^2 (^) f=.fe+l 


= m G (Global mean value) 




Optimum Global Thresholding Using Otsu's 
Method 


Be tween-class variance, < j \ is defined as 





Otsu's Algorithm: Summary 

1. Compute the normalized histogram of the input 
image. Denote the components of the histogram by 
Pi, i=0,1,.... L-l. 

2. Compute the cumulative sums, P^k), for k = 0, 1, ..., 
L-l. 

3. Compute the cumulative means, m(k), for k = 0, 1, ..., 
L-l. 

4. Compute the global intensity mean, m G . 

iA4> 

5. Compute the between-class variance, for k = 0, 1, ..., 
L-l. 

6. Obtain the Otsu's threshold, k*. 

7. Obtain the separability measure. 


Thresholding with Otsu’s method 



Griytov*! bnag* 



3 thresholds with 4 classes 


S«gmnt*<f knag* 
















Binary segmentation - foreground vs. background 


Number of pixels 


Number of pixels 




Foreground 


Threshold between two populations 


Threshold at given percentile 








Binary thresholding - Object detection 




Thermal image 


Thresholded image (Otsu’s method) 


Global threshold selection • threshold too low for detection of the object of interest 






Manual thresholding 



Medium threshold 


High threshold 




Local thresholding 



Threshold computed from graylevel statistics in selected window (Otsu’s method) 



Local thresholding using edge information 



Threshold = average 
graylevel along edges 



Edge image (Canny edge detector 
applied to selected window) 



Thresholded window 





Object detection in video sequences (visible light) 



Change detection 

Absolute difference image 
(Current image - time 
averaged background image) 

Thresholding of difference 
image, i.e. Otsu’s method 

Requires fixed camera (or 
registration of images) 


Daylight video frame 


Thresholded difference image 



Region-Based Segmentation 


Region Growing 

1. Region growing is a procedure that groups pixels or subregions into larger regions. 

2. The simplest of these approaches is pixel aggregation, which starts with a set of "seed" points and from 
these grows regions by appending to each seed points those neighboring pixels that have similar properties 
(such as gray level, texture, color, shape). 

3. Region growing based techniques are better than the edge-based techniques in noisy images where edges 
are difficult to detect. 

Example: Region Growing based on 8-connectivity 

f(x, y ) : input image array 

S(x, y): seed array containing Is (seeds) and Os 
Q(x, y): predicate 


Segmentation by clustering 


• Cluster together (pixels, tokens, etc.) 
that belong together 

• Agglomerative clustering 

- attach closest to cluster it is closest to 

- repeat 

• Divisive clustering 

- split cluster along best boundary 

- repeat 


Point-Cluster distance 

- single-link clustering 

- complete-link clustering 

- group-average clustering 

Dendrograms 

- yield a picture of output as clustering 
process continues 


Clustering Problem Statement 

• Using centroid to represent the huge numbers of clusters 

• Hierarchical clustering, we can change the number of cluster anytime during process if we 
want. 

• Partitional clustering, we have to decide the number of clustering we need first before we 
begin the process. 

• Given a set of vectors {x k ; 1 < k < K}, find a set of M clustering centers {w(i); 1 < i < c} 
such that each x k is assigned to a cluster, say, w(i*), according to a distance (distortion, 
similarity) measure d(x k , w(i)) such that the average distortion 

K j= \ k= i 

is minimized. 

• l(x k ,i) = 1 if x is assigned to cluster i with cluster center w(l); and = 0 otherwise -- indicator 
function. 


Hierarchical clustering 

Algorithm of hierarchical agglomeration(built) • 

A. See every single data as a cluster c,. 


B. Find out c . , c , for the distance is the shortest. 

C. Repeat the steps until satisfies our demand. 


d(a,fr) as the distance between data a and b 


Algorithm of hierarchical division (break up ) • 

• Diameter of cluster 


D(C ,) = 



(a) original image as a cluster C 



and to repeat stpet t) 10 stepd' 


max(J(a,b)) ?for VaeC. , Vb e C. 


ci 



(b)use stepb and stepc to find 
out Cl and Ci clustering. 




(<0use stepd to looking for which 
point is closer to Cl than Cii 


The end of process 










1) See the whole database as one cluster. 

2) Find out the cluster having the biggest diameter 

3) max (d(x,c)) , for VieC. 

4) Split out x as a new cluster , and see the rest data 
points as C i. 

5 ) If d(y, C. ) > d(y, cy, for Vy e c. then split y out of ^ and 
classify it to q 

6) Back to step2 and continue the algorithm until q 
and C. is not change anymore. 



Simple clustering algorithms 


Algorithm 15 + 3: Aggjomerative clustering, or clustering by merging 


Make each point a separate cluster 
Until the clustering is satisfactory' 

Merge the two clusters with the 
smallest in ter-cluster distance 

e n d 


Algorithm 15 + 4: Divisive clustering, or clustering by splitting 


Construct a single cluster containing all points 
Until the clustering is satisfactory 

Split the cluster that yields the two 

components with the largest in ter-cluster distance 

end 






K-Means 


• Choose a fixed number of clusters 


• Choose cluster centers and point- 
cluster allocations to minimize 
error 

• can't do this by search, because 
there are too many possible 
allocations. 


Algorithm 

— fix cluster centers; allocate points to 
closest cluster 

— fix allocation; compute best cluster 
centers 

x could be any set of features for 
which we can compute a distance 
(careful about scaling) 


Image Segmentation by K-Means 


• Select a value of K 

• Select a feature vector for every pixel (color, texture, position, or combination 
of these etc.) 

• Define a similarity measure between feature vectors (Usually Euclidean 
Distance). 

• Apply K-Means Algorithm. 

• Apply Connected Components Algorithm. 

• Merge any components of size less than some threshold to an adjacent 
component that is most similar to it. 


K-Means Clustering 



Image 


Clusters on intensity 


Clusters on color 


K-means clustering using intensity alone and color alone 





K-means (parametric) clustering 


1. Select K points (for example randomly) as 

initial cluster centers 

2. Assign each sample to nearest cluster 

center 

3. Compute new cluster centers (i.e. sample 

means) 






4. Repeat steps 2 and 3 until no further re¬ 
assignments are possible. 


Unlabeled dataset 



Partitional clustering 


• Decide the numbers of the cluster(k-means) 



• Problem: 

• Initial problem 





K-means clustering 



Initial cluster centers (red, green and blue points) 



Samples assigned to nearest cluster center 




K-means clustering 



Re-computed cluster centers 



Samples re-assigned to new cluster centers 




K-means clustering 



Re-computed cluster centers 



Final clustering 





K-means clustering using color 



Original image 



Clustered image - 10 clusters 









K-means Clustering 


Partition the data points into K clusters randomly. Find the centroids of each 
cluster. 

For each data point: 

- Calculate the distance from the data point to each cluster. 

- Assign the data point to the closest cluster. 

Recompute the centroid of each cluster. 

Repeat steps 2 and 3 until there is no further change in the assignment of data 
points (or in the centroids). 


K-Means Clustering 




K-Means Clustering 




K-Means Clustering 




K-Means Clustering 




K-Means Clustering 




K-Means Clustering 


x 




K-Means Clustering 


X 




K-Means Clustering 




k-means Clustering Algorithm 


Initialization: Initial cluster center w(i); 1 < i < c, D(-1 )= 0, l(x k ,i) = 0,1 < i < c, 1 < k < K 
Repeat 

(A) Assign cluster membership (Expectation step) 

Evaluate d(x k , w(i)); 1 < i < c, 1 < k < K 

l(V) = 1 if d(x k , w(i)) < d(x k , w(j)), j * i; 

= 0; otherwise. 1 < k < K 

(B) Evaluate distortion D: 

N 

D(iter ) = / ( x t , i)d(x k , w(i)) 1 < k < K 

k=l 

(C) Update code words according to new assignment (Maximization) 

W(i) = jri( Xk ,i) Xk , N, =J^I(x k ,i), 1 <i<c 

k= 1 k =1 

(D) Check for convergence 

if 1 —D(lter—1 )/D(lter) < e , then convergent = TRUE, 


Mean shift (non-parametric) segmentation 


• Segmentation by clustering of the pixels in the 

image (colour and position) 

• Non-parametric method (Parzen window 

technique) to find modes (i.e. peaks) in the 
density function 

• All pixels climbing to the same peak are 

assigned to the same region. 



(Szeliski: Computer Vision - Algorithms and Applications) 



Mean shift segmentation 



(Szeliski: Computer Vision - Algorithms and Applications) 
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IZED DENSITY 


Mean shift segmentation 



(Szeliski: Computer Vision - Algorithms and Applications) 
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Mean Shift Algorithm 

Mean Shift Algorithm 

1. Choose a search window size. 

2. Choose the initial location of the search window. 

3. Compute the mean location (centroid of the data) in the search window. 

4. Center the search window at the mean location computed in Step 3. 

5. Repeat Steps 3 and 4 until convergence. 


The mean shift algorithm seeks the "mode" or point of highest density of a data distribution: 
















Mean Shift Segmentation 

Mean Shift Setmentation Algorithm 

1. Convert the image into tokens (via color, gradients, texture measures etc). 

2. Choose initial search window locations uniformly in the data. 

3. Compute the mean shift window location for each initial position. 

4. Merge windows that end up on the same "peak" or mode. 

5. The data these merged windows traversed are clustered together. 



*lmage From: Dorin Comaniciu and Peter Meer, Distribution Free Decomposition of Multivariate Data, Pattern Analysis & Applications 
( 1999 ) 2 : 22-30 









































Mean Shift Segmentation Extension 

Is scale (search window size) sensitive. Solution, use all scales: 

Gary Bradski’s internally published aqqlomerative clustering extension: 

Mean shift dendrograms 

1. Place a tiny mean shift window over each data point 

2. Grow the window and mean shift it 

3. Track windows that merge along with the data they transversed 

4. Until everything is merged into one cluster 


Best 4 clusters: Best 2 clusters: 



Advantage over agglomerative clustering: Highly parallelizable 
























Mean Shift Segmentation 



http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html 



























Road segmentation for autonomous vehicles 



Image data (graylevel, colour, local texture) from Pixels with sufficiently high probability density with 
trapezoidal region is used to build a Gaussian respect to the model are assigned to the road class 
model of the road surface. (marked in green). 





Road segmentation - alternative approach 



Original RGB image converted to an illumination 
invariant colour space (reduced variation due to 
sunlight and shadows). From this image a local 
entropy image is derived (Matlab: entropyfilt). 


Segmentation by region growing of the local entropy 
image (Matlab: grayconnected) using the green dots 
(left image) as seed pixels. 




Morphological operations 


• Non-linear filtering 

• Typically used to clean up binary images 

• Erosion: replace pixel value with minimum 

in local neighborhood 

• Dilation: replace pixel value with maximum 

in local neighborhood 

• Structuring element used to define the local 

neighborhood: 
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A shape (in blue) and its morphological dilation (in 
green) and erosion (in yellow) by a diamond- 
shape structuring element. 







Morphological operations - Erosion 



Structuring element (disk shaped) 







Morphological operations - Dilation 



Structuring element (disk shaped) 








Opening = Erosion + Dilation 














Closing = Dilation + Erosion 












Opening - example 



Thresholded image 


Result of opening 


Disk shaped structuring element with radius = 2 pixels (5x5 filter mask) 



Closing - example 



Thresholded image 


Result of opening 


Disk shaped structuring element with radius = 2 pixels (5x5 filter mask) 



Active contours 


Fitting of curves to object boundaries: 

• Snakes (fitting of spline curves to strong 

edges) 

• Intelligent scissors (interactive specification 

of curves clinging to object boundaries) 

• Level set techniques (evolving boundaries 

as the zero set of a characteristic function). 


(Szeliski: Computer Vision - Algorithms and Applications) 




Split and merge methods 


Principles: 

• Recursive splitting of the image based on region 

statistics 

• Hierarchical merging of pixels and regions 

• Combined splitting and merging 

Methods: 

• Watershed segmentation 

• Region splitting (divisive clustering) 

• Region merging (agglomerative clustering) 

• Graph-based segmentation 



(Szeliski: Computer Vision - Algorithms and Applications) 



Split 

The opposite approach to region growing is region splitting. 

It is a top-down approach and it starts with the assumption that the entire 
image is homogeneous 

If this is not true, the image is split into four sub images 


This splitting procedure is repeated recursively until we split the image into 
homogeneous regions 


Split 

• If the original image is square N x N, having dimensions that are powers of 
2(N = 2n) 


• All regions produced but the splitting algorithm are squares having 
dimensions M x M , where M is a power of 2 as well. 


• Since the procedure is recursive, it produces an image representation that 
can be described by a tree whose nodes have four sons each 


Such a tree is called a Quadtree. 


Split 


Quadtree 



R01 


R02 


R04 












Split 

• Splitting techniques disadvantage, they create regions that may be 
adjacent and homogeneous, but not merged. 

• Split and Merge method is an iterative algorithm that includes both 
splitting and merging at each iteration: 


Split / Merge 

• If a region R is inhomogeneous (P(R)= False) then is split into four 
sub regions 


• If two adjacent regions Ri,Rj are homogeneous (P(Ri U Rj) = TRUE), 
they are merged 

• The algorithm stops when no further splitting or merging is possible 


The split and merge algorithm produces more compact regions than 
the pure splitting algorithm 


Region Splitting and Merging 


R: entire image R t : entire image Q: predicate 


1. For any region R t , If Q(R.) = FALSE, 
we divide the image R ; into quadrants. 

2. When no further splitting is possible, 
merge any adjacent regions R and R k 

1/ 

for which Q(Rj u RJ = TRUE. 

3. Stop when no further merging is possible. 


Region Splitting and Merging 


Ri 

R 2 

R 3 

R 4 i 

R42 

r 43 

R44 



a 


FIGURE 10.52 

(a) Partitioned 
image. 

(b) 

Corresponding 
quadtree. R 
represents the 
entire image 
region. 










Normalized cuts 



Separation of groups with weak affinities 
(similarities) between nearby pixels 



(Szeliski: Computer Vision - Algorithms and Applications) 




Graph cuts 



(Szeliski: Computer Vision - Algorithms and Applications) 


Energy-based methods for binary 
segmentation: 

• Grouping of pixels with similar statistics 

• Minimization of pixel-based energy function 

• Region-based and boundary-based energy 

terms 

• Image represented as a graph 

• Cutting of weak edges, i.e. low similarity 

between corresponding pixels. 






